Image processing apparatus and method

ABSTRACT

An image processing apparatus includes a first processor configured to obtain, from a color image, an illumination element image and an albedo element image corresponding to the color image, and a second processor configured to divide the illumination element image into a plurality of subelement images each corresponding to the color image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 USC § 119(a) of KoreanPatent Application No. 10-2016-0143365 filed on Oct. 31, 2016, in theKorean Intellectual Property Office, the entire disclosure of which isincorporated herein by reference for all purposes.

BACKGROUND 1. Field

The following description relates to an image processing apparatus andmethod for dividing an illumination element included in a color imageinto a plurality of subelements.

2. Description of Related Art

Removing and dividing an albedo element from a color image is used toextract illumination information through intrinsic decomposition fromthe color image.

Removing an albedo element from a color image is also used to obtain ashading element image. The shading element image includes a plurality ofundivided elements, for example, a shadow element, a highlight element,and a specular element, indicating similar features in a color space.

However, when a three-dimensional (3D) image is rendered using a relatedshading element image including a plurality of elements, a geometricedge may be present on a portion of a shadow boundary. The shadowelement may be used to estimate a position of the light source, but itis difficult to use the shading element.

Thus, an image processing apparatus and method may be needed toeffectively divide a color image into a plurality of subelement imagesassociated with various elements used for rendering.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

In one general aspect, a system includes an image processing apparatusincluding a first processor configured to obtain, from a color image, anillumination element image and an albedo element image corresponding tothe color image, and a second processor configured to divide theillumination element image into subelement images each corresponding tothe color image.

The image processing apparatus may further include a memory configuredto store instructions. The first processor may be further configured toexecute the instructions to obtain, from the color image, theillumination element image and the albedo element image corresponding tothe color image. The second processor may be further configured toexecute the instructions to divide the illumination element image intothe subelement images each corresponding to the color image.

The second processor may be configured to divide the illuminationelement image into the subelement images using a depth imagecorresponding to the color image. The second processor may be configuredto calculate a normal vector corresponding to each pixel in at least onearea of the color image using the depth image and to divide theillumination element image into the subelement images based on a normalvector change rate between neighboring pixels in the at least one area.

The second processor may be configured to divide the illuminationelement image into the subelement images based on a result of comparingan illumination element change rate between the neighboring pixels tothe normal vector change rate between the neighboring pixels.

The second processor may be configured to divide an illumination elementchange between the neighboring pixels into a shading element and ashadow element based on a degree of correlation between the illuminationelement change rate between the neighboring pixels and the normal vectorchange rate between the neighboring pixels.

The second processor may be configured to calculate a spherical harmonicfunction indicating a corresponding relationship between the normalvector and a shading element from an illumination element change betweenthe neighboring pixels, and to divide the illumination element changebetween the neighboring pixels into the shading element and a shadowelement based on a degree of correlation between the illuminationelement change and the spherical harmonic function.

The subelement images may include any one or combination of a shadingelement image, a shadow element image, a highlight element image, and aspecular element image.

The image processing apparatus may further include a third processorconfigured to estimate a direction of light or a position of at leastone light source of the color image using the shadow element image.

The image processing apparatus may further include a fourth processorconfigured to refine a depth image corresponding to the color imageusing the shading element image.

In another general aspect, a system includes an image processingapparatus including a first processor configured to obtain, from a colorimage, an illumination element image and an albedo element imagecorresponding to the color image, and a second processor configured tocalculate a normal vector corresponding to each pixel of theillumination element image using a depth image corresponding to thecolor image, cluster the illumination element image to at least onepixel group based on a normal vector change rate between neighboringpixels in the illumination element image, and to divide the illuminationelement image into subelement images based on a result of comparing anillumination element change rate between neighboring pixels in the atleast one pixel group to a normal vector change rate of the pixel group.

The image processing apparatus may further include a memory configuredto store instructions. The first processor may be further configured toexecute the instructions to obtain, from the color image, theillumination element image and the albedo element image corresponding tothe color image. The second processor may be further configured toexecute the instructions to calculate the normal vector corresponding toeach pixel of the illumination element image using the depth imagecorresponding to the color image, cluster the illumination element imageto at least one pixel group based on the normal vector change ratebetween neighboring pixels in the illumination element image, and todivide the illumination element image into subelement images based onthe result of comparing the illumination element change rate betweenneighboring pixels in the at least one pixel group to the normal vectorchange rate of the pixel group.

The second processor may be configured to divide an illumination elementchange into a shading element and a shadow element based on a degree ofcorrelation between the illumination element change rate between theneighboring pixels and the normal vector change rate of the pixel group.

The second processor may be configured to calculate a spherical harmonicfunction indicating a corresponding relationship between the normalvector and a shading element from an illumination element change betweenthe neighboring pixels and to divide the illumination element changebetween the neighboring pixels into the shading element and a shadowelement based on a degree of correlation between the illuminationelement change and the spherical harmonic function.

The subelement images may include any one or combination of a shadingelement image, a shadow element image, a highlight element image, and aspecular element image.

In still another general aspect, a system includes an image processingmethod including obtaining, from a color image, an illumination elementimage and an albedo element image corresponding to the color image, anddividing the illumination element image into a plurality of subelementimages each corresponding to the color image.

The dividing of the illumination element image may include calculating anormal vector corresponding to each pixel in at least one area of thecolor image using a depth image corresponding to the color image, anddividing the illumination element image into the subelement images basedon a normal vector change rate between neighboring pixels in the atleast one area.

The dividing of the illumination element image may include dividing anillumination element change between the neighboring pixels into ashading element and a shadow element based on a degree of correlationbetween an illumination element change rate between the neighboringpixels and the normal vector change rate between the neighboring pixels.

The dividing of the illumination element image may include calculating aspherical harmonic function indicating a corresponding relationshipbetween the normal vector and a shading element from an illuminationelement change between the neighboring pixels, and dividing theillumination element change between the neighboring pixels into theshading element and a shadow element based on a degree of correlationbetween the illumination element change and the spherical harmonicfunction.

Other features and aspects will be apparent from the following detaileddescription, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a partial block diagram illustrating an example of an imageprocessing apparatus.

FIG. 2 is a flowchart illustrating an example of an image processingmethod.

FIG. 3 illustrates an example of various color elements included in acolor image.

FIG. 4 illustrates another example of various color elements included ina color image.

FIG. 5 is a flowchart illustrating another example of an imageprocessing method.

Throughout the drawings and the detailed description, unless otherwisedescribed or provided, the same drawing reference numerals will beunderstood to refer to the same elements, features, and structures. Thedrawings may not be to scale, and the relative size, proportions, anddepiction of elements in the drawings may be exaggerated for clarity,illustration, and convenience.

DETAILED DESCRIPTION

The following detailed description is provided to assist the reader ingaining a comprehensive understanding of the methods, apparatuses,and/or systems described herein. However, various changes,modifications, and equivalents of the methods, apparatuses, and/orsystems described herein will be apparent to one of ordinary skill inthe art. The sequences of operations described herein are merelyexamples, and are not limited to those set forth herein, but may bechanged as will be apparent to one of ordinary skill in the art, withthe exception of operations necessarily occurring in a certain order.Also, descriptions of functions and constructions that are well known toone of ordinary skill in the art may be omitted for increased clarityand conciseness.

The features described herein may be embodied in different forms, andare not to be construed as being limited to the examples describedherein. Rather, the examples described herein have been provided merelyto illustrate some of the many possible ways of implementing themethods, apparatuses, and/or systems described herein that will beapparent after an understanding of the disclosure of this application.

The following structural or functional descriptions are exemplary tomerely describe the examples, and the scope of the examples is notlimited to the descriptions provided in the present specification.Various changes and modifications can be made thereto by those ofordinary skill in the art.

Although terms of “first” or “second” are used to explain variouscomponents, the components are not limited to the terms. These termsshould be used only to distinguish one component from another component.For example, a “first” component may be referred to as a “second”component, or similarly, and the “second” component may be referred toas the “first” component within the scope of the right according to theconcept of the present disclosure.

It will be understood that when a component is referred to as being“connected to” another component, the component can be directlyconnected or coupled to the other component or intervening componentsmay be present.

FIG. 1 is a partial block diagram illustrating an example of an imageprocessing apparatus. The image processing apparatus includes an imagereceiver (not shown) and an image processor 100. The image processor 100includes a rendering element analyzer 110, an illumination estimator120, a depth refiner 130, and a renderer 140.

The image receiver receives data required for processing an image. Forexample, the image receiver receives a color image and/or a depth imagecorresponding to the color image. The image receiver includes at leastone processor and at least one hardware component required for receivingand transmitting the data.

The rendering element analyzer 110 divides the color image intosubelement images including various elements to be used for rendering.For example, an illumination element image from which an albedo elementis excluded from the color image and an albedo element image areobtained by dividing the color image. Also, the illumination elementimage may be divided into the subelement images. The rendering elementanalyzer 110 may be implemented by at least one processor. For example,the rendering element analyzer 110 includes a first processor and asecond processor.

The rendering element analyzer 110 may use a feature in a color space toobtain the albedo element image and the illumination element image bydividing the color image. As a non-limited example, in response to achange in color of neighboring pixels in the color image having arelatively high frequency change feature in the color space, therendering element analyzer 110 may identify the change as an intrinsiccolor change of an object which is unrelated to an illumination anddetermine the change as an albedo element. In response to the change inthe colors of the neighboring pixels in the color image having arelatively low frequency change feature in the color space, therendering element analyzer 110 may identify the change as a color changecaused by an illumination effect and determine the change as anillumination element. Based on the aforementioned method, the renderingelement analyzer 110 may obtain the albedo element image and theillumination element image by dividing the color image.

The illumination element image from which the albedo element is excludedmay include a plurality of undivided elements indicating similarfeatures in the color space. For example, the illumination element imageincludes a shading element, a shadow element, a highlight element, and aspecular element. The rendering element analyzer 110 may separate theelements and obtain the subelement images. For example, the renderingelement analyzer 110 divides the illumination element image into ashading element image, a shadow element image, a highlight elementimage, and a specular element image.

Generally, the shading element corresponds to a result obtained by asurface geometry and the shadow element corresponds to a result obtainedwhen light is blocked by another neighboring object. The highlightelement is an opposite concept of the shadow element and indicates thata pixel of the object is relatively bright compared to a neighboringpixel regardless of the surface geometry. The specular element indicatesan influence caused based on a relative position and a relativedirection of a viewpoint and the illumination. That is, the shadingelement is dependent on a change of a normal vector of a surface, butthe shadow element, the highlight element, and the specular element aresubstantially unrelated to the change of the normal vector.

Thus, the rendering element analyzer 110 may use the normal vectorcorresponding to each pixel in the illumination element image to extractthe shading element image from the illumination element image. As anon-limited example, the rendering element analyzer 110 calculates thenormal vector corresponding to each pixel using the depth image receivedby the image receiver and extracts the shading element image from theillumination element image based on a result of comparing anillumination element change rate between the neighboring pixels to anormal vector change rate between the neighboring pixels.

For example, in response to an object having a curved surface forming ashadow, a change rate may be greater than the illumination elementchange rate due to the curved surface. Based on such a feature, theillumination element change rate may be compared to the normal vectorchange rate such that the illumination element change is not identifiedas the shading element on the curved surface. Based on the comparisonresult, the illumination element change occurring during an intervalduring which the illumination element change rate is relatively greatmay be verified to be the shadow element.

In an example, the rendering element analyzer 110 may verify whether theillumination element change between the neighboring pixels is theshading element based on the illumination element change rate betweenthe neighboring pixels and the normal vector change rate between theneighboring pixels. For example, when a similarity between theillumination element change rate between the neighboring pixels and thenormal vector change rate between the neighboring pixels is detectedsuch that a degree of correlation between the illumination elementchange rate and the normal vector change rate is determined to berelatively high, the rendering element analyzer 110 identifies that thecorresponding illumination element change is caused by a surfacegeometry such that the corresponding illumination element change isdetermined to be the shading element.

In another example, when the similarity between the illumination elementchange rate between the neighboring pixels and the normal vector changerate between the neighboring pixels is undetected such that the degreeof correlation between the illumination element change rate and thenormal vector change rate is determined to be relatively low, therendering element analyzer 110 identifies that the correspondingillumination element change is not caused by the surface geometry suchthat the corresponding illumination element change is determined to bethe shadow element.

Also, the rendering element analyzer 110 may use a spherical harmonicfunction indicating a corresponding relationship between a normal vectorand the shading element to extract a shading element image from theillumination element image. For example, based on a feature thatneighboring pixels have similar shading elements when the neighboringpixels in an image have similar normal vector values, a histogram of ashading element value is generated for each normal vector for at leastone area in the image or the spherical harmonic function is calculatedby analyzing the shading element value corresponding to each normalvector. In response to the spherical harmonic function being calculated,the corresponding illumination element change may be determined to bethe shadow element in response to a degree of correlation between theillumination element change and the spherical harmonic function beingrelatively low, that is, the illumination element change being beyond arange of the spherical harmonic function.

The rendering element analyzer 110 may detect a brighter area regardlessof the normal vector change rate between the neighboring pixels toextract a highlight element image from the illumination element image.The rendering element analyzer 110 may detect an area in which a ratiobetween the illumination element change rate and the normal vectorchange rate is rapidly increased or decreased in order to detect aspecular element image from the illumination element image. Here, adifference between a position of a left image and a position of a rightimage or a difference between positions of temporally different framesmay be used to divide and extract the specular element.

In an example, the rendering element analyzer 110 may detect an area inwhich a ratio between an albedo element change rate and the normalvector change rate is rapidly increased or decreased in order to extracta reflection/refraction element image from an albedo element image.

In another example, the rendering element analyzer 110 clusters theillumination element image to at least one pixel group based on thenormal vector change rate between the neighboring pixels in theillumination element image, and divides the illumination element imageinto a plurality of subelement images based on a result of comparing anillumination element change rate between neighboring pixels in the pixelgroup to a normal vector change rate of the pixel group.

The illumination estimator 120 estimates a direction of light or aposition of at least one light source of the color image using theshadow element image obtained by the rendering element analyzer 110. Theillumination estimator 120 may analyze an effect of an illuminationbased on the position of the light source estimated for each pixel in atleast one area. The illumination estimator 120 may be implemented by atleast one processor and include, for example, a third processor.

The depth refiner 130 refines the depth image corresponding to the colorimage using the shading element image obtained by the rendering elementanalyzer 110. The depth refiner 130 may be implemented by at least oneprocessor and include, for example, a fourth processor.

The renderer 130 may perform rendering based on the effect of theillumination, the position of the light source estimated by theillumination estimator 120, and the depth image refined by the depthrefiner 130. The rendering performed by the renderer 140 may beperformed iteratively to render consecutive images. The renderer 140 maybe implemented by at least one processor.

FIG. 2 is a flowchart illustrating an example of an image processingmethod. The image processing method of FIG. 2 may be performed by theimage processor 100 of FIG. 1.

In operation 210, an image processor receives at least one color imageas an input. One color image may be used for analyzing the color imagebased on each element, but a left image and a right image may be usedfor, for example, an identical scene, in order to enhance an accuracy ofthe analyzing.

In operation 220, the image processor obtains an albedo element imageand the illumination element image by dividing the albedo element imageand the illumination element image from the color image. For example,the image processor divides an albedo element from an illuminationelement using a feature of a frequency change in a color space for acolor change between neighboring pixels in the color image.

In operation 230, the image processor obtains a plurality of subelementimages by dividing the color image from which the albedo element isexcluded, that is, the illumination element image, into the subelementimages. For example, the illumination element image is divided into ashading element image, a shadow element image, a highlight elementimage, and a specular element image.

Based on the proposed image processing method, the subelement images maybe obtained by additionally decomposing the illumination element imagefrom which the albedo element is excluded from the color image. Becausea related intrinsic decomposing method focuses on dividing or removingthe albedo element from the color image, various subelements are presentin an illumination element (or a shading element). However, the proposedimage processing method may obtain individual subelement images byseparating each of the subelements.

FIG. 3 illustrates an example of various color elements included in acolor image. An albedo element, a shading element, and a shadow elementare representative color elements.

The albedo element is a feature that determines a reflectivity of light.The albedo element may vary based on a material of an object, a color ofa surface, and a wavelength of light. Referring to a first area 310 ofFIG. 3, a material difference and a color difference are present betweena surface of a table and a surface of a book such that a color value maybe changed based on differences in albedo elements between neighboringpixels in the first area 310. Generally, the albedo element indicates arelatively high frequency change feature in a color space.

The shading element is a feature represented based on a surface geometryof an object or a background. Referring to a second area 320 of FIG. 3,the color value may be changed based on differences in shading elementsbased on a predetermined change rate in the second area 320 because asurface of a cup is curved. Generally, the shading element indicates arelatively low frequency change feature in the color space.

The shadow element is a feature occurring in an area in which a light isblocked by another neighboring object. Referring to a third area 330 ofFIG. 3, because a shadow of the cup is present on the surface of thetable, the color value may be changed based on differences in shadowelements between neighboring pixels in the third area 330. Generally,the shadow element indicates the relatively low frequency change featurein the color space.

In response to the shadow element being divided and extracted from othercolor elements, a direction 340 of light or a position of a light sourcemay be estimated based on the extracted shadow element. Because theshading element is not generally used for estimating the direction 340of light or the position of the light source, the accuracy of estimatingthe direction 340 of light and the position of the light source may beenhanced in response to the shadow element being additionally dividedfrom an illumination element image.

FIG. 4 illustrates another example of various color elements included ina color image. Because a frequency change feature of an albedo elementis different from that of a shading element and that of a shadow elementin a color space, the albedo element may be divided from the color imagebased on a difference in frequency change features. However, the shadingelement and the shadow element may not be divided from the color imagebased on a frequency change feature because the shading element and theshadow element have similar frequency change features in the colorspace.

To divide the shading element and the shadow element from the colorimage, the shading element is used because the shading element isdependent on a change of a normal vector of a surface and the shadowelement is used because the shadow element is substantially irrelevantto the change of the normal vector of the surface.

For example, referring to a first area 410 of FIG. 4, a rapid change ofa normal vector between neighboring pixels may be detected in aneighborhood of a chair from which an illumination element change isdetected. In this example, the illumination element change is identifiedas the shading element. Referring to a second area 420 of FIG. 4,neighboring pixels on a boundary of a shadow from which the illuminationelement change is detected have similar normal vector values. In thisexample, the illumination element change is identified as the shadowelement.

Thus, the illumination element change between the neighboring pixels maybe divided into the shading element and the shadow element based on adegree of correlation between the illumination element change ratebetween the neighboring pixels and a normal vector change rate betweenthe neighboring pixels.

FIG. 5 is a flowchart illustrating another example of an imageprocessing method. The image processing method of FIG. 5 may beperformed by the image processor 100 of FIG. 1.

In operation 510, an image processor receives at least one color imageas an input. One color image may be used for analyzing the color imagebased on each element, but a left image and a right image may be usedfor, for example, an identical scene, in order to enhance an accuracy.

In operation 520, the image processor obtains an albedo element imageand an illumination element image by dividing the albedo element imageand the illumination element image from the color image. For example,the image processor divides an albedo element from an illuminationelement using a frequency change feature in a color space for a colorchange between neighboring pixels in the color image.

In operation 530, the image processor obtains a plurality of subelementimages by dividing the color image from which the albedo element isexcluded, that is, the illumination element image, into the subelementimages. As described above with reference to FIG. 4, a shadow elementimage and a shading element image may be divided from the color imagebased on a result of comparing an illumination element change ratebetween neighboring pixels to a normal vector change rate between theneighboring pixels.

In operation 540, the image processor estimates a direction of light ora position of at least one light source of the color image using theshadow element image. Also, the image processor may analyze an effect ofan illumination based on the position of the light source estimated foreach pixel in at least one area.

In operation 550, the image processor refines a depth imagecorresponding to the color image using the shading element image.Because the shading element from which the shadow element is divided andexcluded is used, the depth image may be accurately refined.

In operation 560, the image processor performs rendering based on theeffect of the illumination, the position of the light source estimatedin operation 540, and the depth image refined in operation 550. Therendering may be performed iteratively for rendering consecutive images.

Thus, the proposed image processing method may obtain a plurality ofsubelement images by dividing the illumination element image into thesubelement images, such as the shadow element image and the shadingelement image, and use the subelement images for an additional analysisand refinement to enhance a rendering quality.

The image processing apparatus 100, the rendering element analyzer 110,the illumination estimator 120, the depth refiner 130, and the renderer140 in FIG. 1 that perform the operations described in this applicationare implemented by hardware components configured to perform theoperations described in this application that are performed by thehardware components. Examples of hardware components that may be used toperform the operations described in this application where appropriateinclude controllers, sensors, generators, drivers, memories,comparators, arithmetic logic units, adders, subtractors, multipliers,dividers, integrators, and any other electronic components configured toperform the operations described in this application. In other examples,one or more of the hardware components that perform the operationsdescribed in this application are implemented by computing hardware, forexample, by one or more processors or computers. A processor or computermay be implemented by one or more processing elements, such as an arrayof logic gates, a controller and an arithmetic logic unit, a digitalsignal processor, a microcomputer, a programmable logic controller, afield-programmable gate array, a programmable logic array, amicroprocessor, or any other device or combination of devices that isconfigured to respond to and execute instructions in a defined manner toachieve a desired result. In one example, a processor or computerincludes, or is connected to, one or more memories storing instructionsor software that are executed by the processor or computer. Hardwarecomponents implemented by a processor or computer may executeinstructions or software, such as an operating system (OS) and one ormore software applications that run on the OS, to perform the operationsdescribed in this application. The hardware components may also access,manipulate, process, create, and store data in response to execution ofthe instructions or software. For simplicity, the singular term“processor” or “computer” may be used in the description of the examplesdescribed in this application, but in other examples multiple processorsor computers may be used, or a processor or computer may includemultiple processing elements, or multiple types of processing elements,or both. For example, a single hardware component or two or morehardware components may be implemented by a single processor, or two ormore processors, or a processor and a controller. One or more hardwarecomponents may be implemented by one or more processors, or a processorand a controller, and one or more other hardware components may beimplemented by one or more other processors, or another processor andanother controller. One or more processors, or a processor and acontroller, may implement a single hardware component, or two or morehardware components. A hardware component may have any one or more ofdifferent processing configurations, examples of which include a singleprocessor, independent processors, parallel processors,single-instruction single-data (SISD) multiprocessing,single-instruction multiple-data (SIMD) multiprocessing,multiple-instruction single-data (MISD) multiprocessing, andmultiple-instruction multiple-data (MIMD) multiprocessing.

The methods illustrated in FIGS. 2 and 5 that perform the operationsdescribed in this application are performed by computing hardware, forexample, by one or more processors or computers, implemented asdescribed above executing instructions or software to perform theoperations described in this application that are performed by themethods. For example, a single operation or two or more operations maybe performed by a single processor, or two or more processors, or aprocessor and a controller. One or more operations may be performed byone or more processors, or a processor and a controller, and one or moreother operations may be performed by one or more other processors, oranother processor and another controller. One or more processors, or aprocessor and a controller, may perform a single operation, or two ormore operations.

Instructions or software to control computing hardware, for example, oneor more processors or computers, to implement the hardware componentsand perform the methods as described above may be written as computerprograms, code segments, instructions or any combination thereof, forindividually or collectively instructing or configuring the one or moreprocessors or computers to operate as a machine or special-purposecomputer to perform the operations that are performed by the hardwarecomponents and the methods as described above. In one example, theinstructions or software include machine code that is directly executedby the one or more processors or computers, such as machine codeproduced by a compiler. In another example, the instructions or softwareincludes higher-level code that is executed by the one or moreprocessors or computer using an interpreter. The instructions orsoftware may be written using any programming language based on theblock diagrams and the flow charts illustrated in the drawings and thecorresponding descriptions in the specification, which disclosealgorithms for performing the operations that are performed by thehardware components and the methods as described above.

The instructions or software to control computing hardware, for example,one or more processors or computers, to implement the hardwarecomponents and perform the methods as described above, and anyassociated data, data files, and data structures, may be recorded,stored, or fixed in or on one or more non-transitory computer-readablestorage media. Examples of a non-transitory computer-readable storagemedium include read-only memory (ROM), random-access memory (RAM), flashmemory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs,DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, magnetictapes, floppy disks, magneto-optical data storage devices, optical datastorage devices, hard disks, solid-state disks, and any other devicethat is configured to store the instructions or software and anyassociated data, data files, and data structures in a non-transitorymanner and provide the instructions or software and any associated data,data files, and data structures to one or more processors or computersso that the one or more processors or computers can execute theinstructions. In one example, the instructions or software and anyassociated data, data files, and data structures are distributed overnetwork-coupled computer systems so that the instructions and softwareand any associated data, data files, and data structures are stored,accessed, and executed in a distributed fashion by the one or moreprocessors or computers.

While this disclosure includes specific examples, it will be apparentafter an understanding of the disclosure of this application thatvarious changes in form and details may be made in these exampleswithout departing from the spirit and scope of the claims and theirequivalents. The examples described herein are to be considered in adescriptive sense only, and not for purposes of limitation. Descriptionsof features or aspects in each example are to be considered as beingapplicable to similar features or aspects in other examples. Suitableresults may be achieved if the described techniques are performed in adifferent order, and/or if components in a described system,architecture, device, or circuit are combined in a different manner,and/or replaced or supplemented by other components or theirequivalents. Therefore, the scope of the disclosure is defined not bythe detailed description, but by the claims and their equivalents, andall variations within the scope of the claims and their equivalents areto be construed as being included in the disclosure.

What is claimed is:
 1. An image processing apparatus comprising: a firstprocessor configured to obtain, from a color image, an illuminationelement image and an albedo element image corresponding to the colorimage; and a second processor configured to divide the illuminationelement image into subelement images each corresponding to the colorimage.
 2. The apparatus of claim 1, further comprising a memoryconfigured to store instructions; wherein the first processor is furtherconfigured to execute the instructions to obtain, from the color image,the illumination element image and the albedo element imagecorresponding to the color image; and wherein the second processor isfurther configured to execute the instructions to divide theillumination element image into the subelement images each correspondingto the color image.
 3. The apparatus of claim 2, wherein the secondprocessor is further configured to divide the illumination element imageinto the subelement images using a depth image corresponding to thecolor image.
 4. The apparatus of claim 3, wherein the second processoris further configured to calculate a normal vector corresponding to eachpixel in at least one area of the color image using the depth image andto divide the illumination element image into the subelement imagesbased on a normal vector change rate between neighboring pixels in theat least one area.
 5. The apparatus of claim 4, wherein the secondprocessor is further configured to divide the illumination element imageinto the subelement images based on comparing an illumination elementchange rate between the neighboring pixels to the normal vector changerate between the neighboring pixels.
 6. The apparatus of claim 5,wherein the second processor is further configured to divide anillumination element change between the neighboring pixels into ashading element and a shadow element based on a degree of correlationbetween the illumination element change rate between the neighboringpixels and the normal vector change rate between the neighboring pixels.7. The apparatus of claim 4, wherein the second processor is furtherconfigured to calculate a spherical harmonic function indicating acorresponding relationship between the normal vector and a shadingelement from an illumination element change between the neighboringpixels, and to divide the illumination element change between theneighboring pixels into the shading element and a shadow element basedon a degree of correlation between the illumination element change andthe spherical harmonic function.
 8. The apparatus of claim 2, whereinthe subelement images include any one or combination of a shadingelement image, a shadow element image, a highlight element image, and aspecular element image.
 9. The apparatus of claim 8, further comprising:a third processor configured to estimate a direction of light or aposition of at least one light source of the color image using theshadow element image.
 10. The apparatus of claim 8, further comprising:a fourth processor configured to refine a depth image corresponding tothe color image using the shading element image.
 11. An image processingapparatus comprising: a first processor configured to obtain, from acolor image, an illumination element image and an albedo element imagecorresponding to the color image; and a second processor configured tocalculate a normal vector corresponding to each pixel of theillumination element image using a depth image corresponding to thecolor image, cluster the illumination element image to at least onepixel group based on a normal vector change rate between neighboringpixels in the illumination element image, and to divide the illuminationelement image into subelement images based on comparing an illuminationelement change rate between neighboring pixels in the at least one pixelgroup to a normal vector change rate of the pixel group.
 12. Theapparatus of claim 11, further comprising a memory configured to storeinstructions; wherein the first processor is further configured toexecute the instructions to obtain, from the color image, theillumination element image and the albedo element image corresponding tothe color image; and wherein the second processor is further configuredto execute the instructions to calculate the normal vector correspondingto each pixel of the illumination element image using the depth imagecorresponding to the color image, cluster the illumination element imageto at least one pixel group based on the normal vector change ratebetween neighboring pixels in the illumination element image, and todivide the illumination element image into subelement images based onthe result of comparing the illumination element change rate betweenneighboring pixels in the at least one pixel group to the normal vectorchange rate of the pixel group.
 13. The apparatus of claim 12, whereinthe second processor is further configured to divide an illuminationelement change into a shading element and a shadow element based on adegree of correlation between the illumination element change ratebetween the neighboring pixels and the normal vector change rate of thepixel group.
 14. The apparatus of claim 12, wherein the second processoris further configured to calculate a spherical harmonic functionindicating a corresponding relationship between the normal vector and ashading element from an illumination element change between theneighboring pixels and to divide the illumination element change betweenthe neighboring pixels into the shading element and a shadow elementbased on a degree of correlation between the illumination element changeand the spherical harmonic function.
 15. The apparatus of claim 12,wherein the subelement images include any one or combination of ashading element image, a shadow element image, a highlight elementimage, and a specular element image.
 16. An image processing methodcomprising: obtaining, from a color image, an illumination element imageand an albedo element image corresponding to the color image; anddividing the illumination element image into subelement images eachcorresponding to the color image.
 17. The method of claim 16, whereinthe dividing of the illumination element image comprises: calculating anormal vector corresponding to each pixel in at least one area of thecolor image using a depth image corresponding to the color image; anddividing the illumination element image into the subelement images basedon a normal vector change rate between neighboring pixels in the atleast one area.
 18. The method of claim 17, wherein the dividing of theillumination element image comprises dividing an illumination elementchange between the neighboring pixels into a shading element and ashadow element based on a degree of correlation between an illuminationelement change rate between the neighboring pixels and the normal vectorchange rate between the neighboring pixels.
 19. The method of claim 17,wherein the dividing of the illumination element image comprises:calculating a spherical harmonic function indicating a correspondingrelationship between the normal vector and a shading element from anillumination element change between the neighboring pixels; and dividingthe illumination element change between the neighboring pixels into theshading element and a shadow element based on a degree of correlationbetween the illumination element change and the spherical harmonicfunction.
 20. A non-transitory computer-readable medium storinginstructions that, when executed by one or more processors, cause theone or more processors to perform the image processing method of claim16.